Newton-type Methods

Newton’s method is one of the most famous numerical methods. Its origins, as the name suggests, lies in part with Newton, but the form familiar to us today is due to Simpson of “Simpson’s Rule” fame. Originally proposed to find the roots of polynomials it was Simpson who proposed using it for general nonlinear equations and for its use in optimization by finding a root of the gradient. Here we discuss Newton’s method for both of these problems with the focus on the somewhat harder optimization problem. A disadvantage of Newton’s method, in its original form, is that it has only local convergence unless the class of functions is severely restricted. Much of the discussion here will be about the many ways Newton’s method may be modified to achieve global convergence. A key aim of all these methods is that once the iterates become sufficiently close to a solution the method takes Newton steps.